Multimodal Fusion of Electromagnetic, Ultrasound and MRI Data for Building an Articulatory Model

Data fusion from multiple sensors is of significant interest to the speech research community, as it can potentially provide a better picture of speech production through the use of complementary sensor modalities. This paper deals with the practical aspects of this problem, such as acquisition and processing of the dynamic ultrasound (US) and electromagnetic (EM) data of the tongue during speech production, static MRI images of the vocal tract using repetitions, and registration of the data from these different sources to a common reference frame. To the best of our knowledge, this is the first work that demonstrates the potential of static and dynamic data fusion in the construction of articulatory databases.